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Background 

There are two main reasons why textbooks are evaluated; the first which is 
associated to the task of textbook selection (predictive evaluation) and the second 
to the need to evaluate it while in use (retrospective evaluation) so that teachers 
can implement adaptation procedures later. Predictive evaluation is almost non¬ 
existent in most developing countries as textbooks are usually provided free or 
through a textbook loan scheme. There is however hope for getting teachers 
involved in while-use/post use or retrospective evaluation of textbooks as they not 
only help teachers with adaptation but help teachers in their overall professional 
development. Since textbooks are an essential part of the teachers’ professional 
life any form of out-of-class work that involves the teacher’s own knowledge 
building can become an integral part of the teacher’s professional development. 

The two main ways English Language Teaching (ELT) textbooks are evaluated 
are through impression, which involves teacher intuitions and by way of use of an 
instrument, usually a checklist. While the former has been known to be effective 
especially if done by experienced teachers, the latter is more common because in 
some teaching situations evaluations are carried out by several teachers who have 
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to use the same book. In such situations evaluations are best done using a 
common instrument. 

Teachers can easily find checklists for evaluation of ELT textbooks. There are 
numerous checklists available all of which have been developed by individuals or 
institutions taking into consideration their own special needs. While there are 
scores of instruments available, the literature regarding reliability and validity of 
these instruments is lacking. 

The need to redefine textbook evaluation in ELT 

There is a need to redefine textbook evaluation in ELT. The development of 
textbook evaluation instruments from the perspective of predictive evaluation is 
still relevant but what teachers need to do is to look at evaluation from a wider 
perspective - a perspective which encompasses diagnostic, formative and 
developmental aspects of textbook use. It is when evaluation is viewed from this 
perspective that while-use and post-use evaluation of textbooks become more 
important than evaluation for selection. 

If retrospective evaluation of textbooks is to be emphasized then there would be a 
need to evaluate current tools available which have been predominantly developed 
for use for selection purposes. The checklists which have dominated textbook 
evaluation for decades (Mukundan and Ahour, 2010) have been found to be 
sometimes teacher-unfriendly. Some of the evaluation criteria are also found to be 
irrelevant. Many of the items which require evaluators to judge on vocabulary 
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loading and distribution, for instance, defy logic as judging cannot be humanly 
possible under time constraints (Mukundan,2007). 

The concept in the invention of software for textbook evaluation 

Conceptually the software for textbook evaluation (RETROTEXT-E 1.0) is based 
on the Composite Framework for ESL Textbook Evaluation (Mukundan, 2004, 
2006). This framework promotes triangulation of instruments so that evaluation is 
not overly dependent of data from only one source, which has traditionally been 
from the checklist. In this new framework, the checklist is supported by two other 
instruments, the computerized evaluator of vocabulary loading and distribution 
patterns (Word Analyzer) and the teachers’ log. 


The functions within RETROTEXT-E 1.0 

The software was designed to be as user-friendly as possible. The main functions 
of this programme are: 
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Part 1: Word Loading and Distribution Patterns (computerized analyzer) 

Part 2: Overall Checklist Evaluation 
Part 3: Teacher Log 
Part 4: View report 

Before the evaluator begins it would be necessary to load text files (txt) of the 
digitized pages of the textbook into the software so that Part 1 of the evaluation 
can be successfully carried out. 

How the different parts of RETROTEXT-E 1.0 work 

Part 1 

In Part 1, the textbook evaluation programme assists evaluators by providing key 
data on the following: 

1. Is the loading of vocabulary in the book and across chapters pedagogically 
sound? 

2. Do words used in the textbook reflect those in wordlists like the GSL? 

3. Are repetition and recycling of vocabulary emphasized? 

Unlike the mono-instrument instrument, the checklist, Part 1 of the software can 
empirically determine how effective vocabulary loading is as the programme has 
the ability to compute total number of words across the book and in individual 
chapters. The full list of words is also provided. In addition to this analysis of 
words in the book to determine suitability of use from the context of frequency 
can be carried out. The software does comparison of the words in the textbook 
with that of words in the GSL. This to some extent will determine if textbook 
writers have used the most frequently used words in the English language as a 
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starting point, particularly at the lower levels. Another function of Part 1 is to help 
evaluators determine if words introduced are recycled within chapters and across 
the book. This is important as many experts have said that words introduced to 
learners must be repeated at least seven times at intervals so that they can be 
easily remembered (Thombury, 2002). The screen shot of the analysis of Part 1 


will be as in the figure below: 



Part 2 

Part 2 of the software assists evaluators by way of a checklist to determine the 
efficiency of the book while in use. The checklist feature was retained as the 
developer felt that inventions need not replace old but essentially economical 
aspects of past inventions. The checklist can help teacher evaluators provide a 
quick appraisal of the textbook immediately after a lesson, after a week’s 
teaching, after a unit or any other time interval. Data from Evaluations at intervals 
can be saved for study later. 
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Part 3 

Part 3 is the teachers’ log where teachers can record anything to do with textbook 
use in classroom teaching. Part 3 has three sub-parts: 


Sub-part A 

The teacher can do an overall evaluation of the textbook (after teaching) by 
indicating with scores how well it does in terms of “role” and how it performs 
alongside the other “roles” - that of teacher and learner. 
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Sub-part B 

Reflection - the teacher uses this to record observations, feelings, etc (this much 
like a conventional journal or diary. 

Sub-part C 

Suggestions for Adaptation - in this part, teachers can put down ideas that they 
have on what would work better. Teachers can record the page numbers and 
activity numbers for easy referencing later. 
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Part 4 

This is the Reports Section which keeps data gathered from parts 1-3. These 
reports can be retrieved at any time by the teacher. 



277 



























































































The advantages RETROTEXT-E 1.0 have over other types of textbook 
evaluation 

The main difference between RETROTEXT-E 1.0 and the competition is that 
when compared to intuitive methods it is empirical and when compared to 
traditional methods which use instruments like the checklist it is superior in terms 
of reliability (Mukundan, 2004). Also, traditional checklists are developed for 
predictive evaluation (mainly for selection purposes). RETROTEXT-E 1.0 while 
it can still perform as a framework for predictive evaluation is more suited for 
retrospective evaluation. The other advantages of this software are: 

i) The software provides record-keeping facilities which are very useful 
if retrospective evaluation of textbooks is going to be an important 
feature in the life of teachers. Record keeping is user-friendly and the 
fact that this is computerized minimizes the threat of loss of notes 
when it is done using conventional pen and paper sources. 

ii) Evaluation can be daily, weekly, monthly, quarterly, yearly - the 
teacher is in-charge of his or her own evaluation. Data are recorded 
with dates and when printed out this will be useful for monitoring 
evaluations from end-to-end. 

iii) Evaluations can be done individually or in teams. Reports by a team of 
evaluators can be printed and used as documents for analysis at panel 
meetings. 

iv) Teacher professional development is enhanced because the evaluations 
done can be a source for discussion at end of year teacher workshops 
where teachers teaching the same level and using the same book can 
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deliberate on suggestions on adaptation made by different teachers and 
come up with activities that can be replacements for those deemed 
unfit for use. 

Conclusion 

This innovation is probably not the answer to all the problems that confront 
teachers with regards textbook evaluation. However, because it does not 
altogether have a summative stance in evaluation and it supports retrospective 
evaluation of textbooks, it can become a useful tool for the teacher for not only 
evaluating textbooks but researching them. 
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